A Temporal Minimum Description Length Policy for Evolving Neural Networks

One of the most important issues for computational methods is their time complexity. This paper introduces a temporal MDL (minimum description length) policy for evolving neural networks based on their execution time on the hosting hardware. Temporal MDL implements an adaptive selection pressure based on the actual processing time of the evolving solutions and thus favors creation of faster, more compact networks for the given data. Temporal MDL reduces the time complexity directly and the network size indirectly. The latter helps generalization and reduces model variance, making the temporal MDL a viable candidate for regularization. This methodology is especially important for time-critical applications. Mackey-Glass time series prediction results are presented for evolutionary distributed time lag neural networks with temporal MDL to demonstrate the above stated capabilities. INTRODUCTION Bias-variance dilemma is an important issue in neural network design. Besides validation-based early stopping, regularization is an effective method for helping networks’ generalization capabilities by penalizing more complex solutions and reducing unwanted variance. This is especially important when the training data is scarce and one does not have the luxury of setting aside a part of data for validation-based early stopping, which is usually the case for many real world applications (Haykin, 1999). Speaking of real world applications, time is usually one of the most important factors in computing systems, especially for time-critical applications. Furthermore, temporal agility has been mentioned as an indicator of machine intelligence (Kurzweil, 2000). Traditional regularization techniques such as Akaike information criterion (Box et al, 1994) do not deal with the actual time complexity. Here we introduce a practical minimum description length procedure in the context of evolutionary neural networks that will address this issue. TEMPORAL MINIMUM DESCRIPTION LENGTH AND NEURAL NETWORKS Our hypothesis is as follows: since the actual training time for a neural network on a given computational platform is directly related to its size, then penalizing each network not only for its error but also its time complexity should reduce its space complexity which has been reported by other researchers as a successful form of regularization (Principe et al, 2000; Sathyanarayan and Kumar, 1996; Hansen and Yu, 2001). This favoring of parsimony through selection pressure on both error and actual computation